test: ignore stale AI agents init prompts during sample download by v1212 · Pull Request #9102 · Azure/azure-dev

v1212 · 2026-07-13T02:27:54Z

Fixes #9101

Summary

Updates the AI agents Tier 2 live E2E console driver to ignore answered survey prompts once azd init progress appears after them.
Prevents false init stuck in prompt loop failures when survey leaves an answered starter/agent-template prompt visible on the PTY screen during sample download/copy.
Fails fast on invalid GitHub token / GitHub CLI authentication prompts instead of retrying into gh browser/device login in CI.

Root cause

The live test driver identifies the active interactive prompt by scanning the rendered virtual-terminal screen for prompt lines.
After selecting a starter template, survey can leave the answered ? select a starter template line on screen while azd init continues downloading and copying the sample.
In slower/timing-sensitive runs, the screen can contain both the stale prompt and later init progress such as Downloading sample from GitHub, azure.yaml, or src/....
The old driver treated that stale prompt as still active, retried prompt handling even though the CLI was no longer waiting for input, and eventually failed with a prompt-loop error.
PR live runs also showed that pipeline-provided GH_TOKEN / GITHUB_TOKEN can be invalid. Retrying without validating token state lets gh fall back to interactive auth prompts or fail later during sample adoption.
GitHub CLI device-login output can end on a non-? line, so the driver must classify auth/device-login lines explicitly instead of waiting for a survey prompt.

Fix details

activePrompt now treats answered prompts followed by init progress as stale.
Answered starter/agent-template prompts without active choice rows are ignored instead of re-dispatched.
GitHub CLI auth prompts and device-login lines are detected explicitly and fail with a clear token/auth error.
The live driver strips inherited GitHub token env vars, validates any candidate token before passing it to azd, and avoids poisoning later test modes with an unusable token.
phaseInitWithRetry was renamed to phaseInitWithTokenCheck to reflect that it now classifies token failures rather than retrying.
Unit coverage was added for stale prompt detection, later real prompts after progress, GitHub auth prompt classification, and non-? GitHub device-login lines.

Why this surfaced now

ci: associate AI agents live pipeline with main PRs #9039 did not introduce the driver bug; it associated the live pipeline with relevant PRs so /azp run azure-dev - live - ext - azure.ai.agents can start PR comment-triggered runs.
The PR-triggered path exposed pre-existing PTY/rendering timing races and token/auth retry behavior that did not consistently appear in earlier manual/main runs.

Validation

Local:

gofmt -w tests/e2e-live/console_test.go tests/e2e-live/tier2_live_test.go
go test -count=1 ./tests/e2e-live/
GOOS=linux GOARCH=amd64 GOTOOLCHAIN=go1.26.4 go fix -diff ./...
GOOS=linux GOARCH=amd64 GOTOOLCHAIN=go1.26.4 go test -c ./tests/e2e-live/

CI / live:

Public PR pipeline passed on latest commit 77a867d4: build 6552884.
All GitHub checks passed on latest commit 77a867d4.
All review threads are resolved.
Comment-triggered live pipeline passed on latest commit 77a867d4: build 6552933.
A second consecutive comment-triggered live pipeline passed on latest commit 77a867d4: build 6552980.
Earlier successful live runs before later review-fix commits: 6551457 and 6551564.
Earlier failing runs that motivated fixes: 6551005 (stale answered template prompt), 6551073 (invalid GitHub token retry entered gh browser auth), and 6552245 (invalid inherited token reused in the next mode).

github-actions · 2026-07-13T02:28:14Z

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag @RickWinter and @kristenwomack to let us know.

v1212 · 2026-07-13T02:30:17Z

/azp run azure-dev - live - ext - azure.ai.agents

azure-pipelines · 2026-07-13T02:30:31Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR updates the Azure AI Agents Tier 2 live E2E console driver to avoid misclassifying already-answered Survey prompts as “active” when init progress output appears after them (e.g., during slower GitHub sample downloads), preventing false “prompt loop” failures.

Changes:

Extend prompt detection to ignore ? prompt lines when init progress output is present after the prompt.
Add init-progress heuristics that recognize GitHub sample download/copy/adoption output lines.

jongio

The new hasInitProgressAfterPrompt trips the required lint / go-fix check, which is currently red and blocking merge. The file is build-tagged //go:build linux, so go fix only flags it on Linux (a local Windows run passes, which is easy to miss). Running go fix ./... with GOOS=linux in cli/azd/extensions/azure.ai.agents rewrites the manual loop to slices.ContainsFunc and adds the slices import.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

jongio

The go-fix rewrite I flagged is in: hasInitProgressAfterPrompt now uses slices.ContainsFunc with the slices import, and the required lint / go-fix check is green. Approving to clear my earlier comment.

v1212 · 2026-07-13T03:24:45Z

/azp run azure-dev - live - ext - azure.ai.agents

azure-pipelines · 2026-07-13T03:24:56Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

jongio

The new commit adds isAnsweredTemplatePrompt and two focused unit tests for activePrompt (answered select without choices, active select with choices). The lint / go-fix check I flagged earlier is green. Re-approving against the current HEAD (2d1c700).

v1212 · 2026-07-13T04:02:52Z

/azp run azure-dev - live - ext - azure.ai.agents

azure-pipelines · 2026-07-13T04:03:04Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

v1212 · 2026-07-13T08:48:51Z

/azp run azure-dev - live - ext - azure.ai.agents

azure-pipelines · 2026-07-13T08:49:11Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

v1212 · 2026-07-13T11:44:50Z

/azp run azure-dev - live - ext - azure.ai.agents

azure-pipelines · 2026-07-13T11:45:22Z

Azure Pipelines successfully started running 1 pipeline(s).

v1212 · 2026-07-13T12:10:47Z

/azp run azure-dev - live - ext - azure.ai.agents

azure-pipelines · 2026-07-13T12:10:58Z

Azure Pipelines successfully started running 1 pipeline(s).

jongio

Reviewed the latest test-driver changes against 77a867d. The runner now strips unusable GITHUB_TOKEN/GH_TOKEN from its env and validates a token with gh auth status before using it, and init still fails fast when it reaches an interactive GitHub CLI auth prompt, including the non-? device-login line. The withoutGitHubTokenEnv helper allocates a fresh slice instead of aliasing os.Environ(). Covered by the new activePrompt and isGitHubAuthPrompt unit tests. Re-approving against the current HEAD.

vhvb1989 · 2026-07-13T18:54:56Z

/check-enforcer evaluate

trangevi

Decided that my questions don't need to be considered blocking

Ignore stale init prompts during sample download

9fd6cfa

Copilot AI review requested due to automatic review settings July 13, 2026 02:27

v1212 requested review from JeffreyCA, glharper, huimiu, therealjohn, trangevi and trrwilson as code owners July 13, 2026 02:27

Copilot started reviewing on behalf of v1212 July 13, 2026 02:28 View session

microsoft-github-policy-service Bot assigned v1212 Jul 13, 2026

github-actions Bot added the ext-agents azure.ai.agents extension label Jul 13, 2026

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/console_test.go Outdated

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/console_test.go

jongio reviewed Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/console_test.go

test: apply go fix for stale init prompt helper

d90ffe8

Copilot AI review requested due to automatic review settings July 13, 2026 03:00

Copilot started reviewing on behalf of v1212 July 13, 2026 03:00 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/console_test.go Outdated

jongio approved these changes Jul 13, 2026

View reviewed changes

test: ignore answered template prompts without choices

2d1c700

Copilot AI review requested due to automatic review settings July 13, 2026 03:42

Copilot started reviewing on behalf of v1212 July 13, 2026 03:42 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/console_test.go

jongio approved these changes Jul 13, 2026

View reviewed changes

test: detect GitHub device login prompt

4306afe

Copilot AI review requested due to automatic review settings July 13, 2026 08:33

Copilot started reviewing on behalf of v1212 July 13, 2026 08:33 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

test: ignore unusable GitHub tokens in live driver

bf3d1de

Copilot AI review requested due to automatic review settings July 13, 2026 09:16

Copilot started reviewing on behalf of v1212 July 13, 2026 09:17 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/tier2_live_test.go Outdated

test: continue past invalid GitHub token candidates

bcfb34e

Copilot AI review requested due to automatic review settings July 13, 2026 11:18

Copilot started reviewing on behalf of v1212 July 13, 2026 11:21 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/tier2_live_test.go Outdated

test: validate stored GitHub token fallback

77a867d

Copilot AI review requested due to automatic review settings July 13, 2026 11:28

Copilot started reviewing on behalf of v1212 July 13, 2026 11:30 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

jongio approved these changes Jul 13, 2026

View reviewed changes

vhvb1989 approved these changes Jul 13, 2026

View reviewed changes

vhvb1989 enabled auto-merge (squash) July 13, 2026 18:54

trangevi requested changes Jul 13, 2026

View reviewed changes

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/console_test.go

Comment thread cli/azd/extensions/azure.ai.agents/tests/e2e-live/tier2_live_test.go

trangevi approved these changes Jul 13, 2026

View reviewed changes

vhvb1989 merged commit 7eed332 into main Jul 14, 2026
32 checks passed

v1212 mentioned this pull request Jul 14, 2026

Improve AI agents live PTY prompt hardening #9132

Open

Uh oh!

Conversation

v1212 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Fix details

Why this surfaced now

Validation

Uh oh!

github-actions Bot commented Jul 13, 2026

📋 Prioritization Note

Uh oh!

v1212 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

jongio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jongio left a comment

Choose a reason for hiding this comment

Uh oh!

v1212 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jongio left a comment

Choose a reason for hiding this comment

Uh oh!

v1212 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

v1212 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

v1212 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

v1212 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

v1212 commented Jul 13, 2026 •

edited

Loading